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ABSTRACT 


This  paper  explores  the  possibility  of  applying  multi-sensor  information  fusion  technology  to  the 
development  of  smart  aircraft.  This  technology  integrates  information  from  multiple  sensors  and 
extracts  tactical  information  to  detect,  track  and  identify  time  critical  targets  at  any  time,  in  any 
place  and  under  all  weather  conditions.  Such  target  information  will  help  the  war  fighter  avoid 
fratricide  and  deploy  weapon  systems  for  surgical  strike.  Finally,  the  smart  aircraft  helps  the  war 
fighter  establish  air  superiority  in  the  air,  on  the  land  and  at  sea. 

The  architecture  of  a  smart  aircraft  includes  the  pilot  and  flight  crew,  vehicle  interface  unit, 
multi-sensor  correlation  processor,  multi-sensor  image  fusion  model,  multi-sensor  track  fusion 
model,  multiple  target  identification  fusion  model  and  multiple  sensors  such  as:  Radar,  CNI, 
EW,  IFF  Visual,  Electro  Optic  and  IR. 

The  multi-sensor  track  fusion  model  computes  a  fused  track  from  the  sensor  trackers.  A 
Multiple  Target  Identification  Fusion  model  creates  integrated  target  identifications  from 
identification  of  multiple  targets.  Similarly,  a  Multi-Sensor  Image  Fusion  model  creates  an 
integrated  target  image  from  multiple  sensor  images. 

In  traditional  aircraft,  the  pilot  and  crew  extract  tactical  information  from  a  huge  amount  of 
multi-sensor  information.  This  processing  is  time  consuming  and  subject  to  human  error.  A 
future  smart  aircraft  with  Multi-Sensor  Information  Fusion  technology  will  have  access  to 
accurately  correlated  fused  information.  Fused  information  assists  the  pilot  and  crew  to  detect, 
track  and  identify  targets  more  rapidly  and  accurately.  Such  platforms  are  truly  smart  aircraft. 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1.  REPORT  DATE 

SEP  2002 

2.  REPORT  TYPE 

3.  DATES  COVERED 

00-00-2002  to  00-00-2002 

4.  TITLE  AND  SUBTITLE 

Multi- Sensor  Information  Fusion  Technology  Applied  to  the 

Development  of  Smart  Aircraft 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Lockheed  martin  Aeronautics  Company, Marietta, GA, 30060 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR'S  ACRONYM(S) 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

The  original  document  contains  color  images. 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 
ABSTRACT 

18.  NUMBER 
OF  PAGES 

10 

19a.  NAME  OF 
RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


1  INTRODUCTION 


The  smartness  of  the  fighter  aircraft  can  be  characterized  by  the  following  qualities: 

(1)  Quickly  assess  the  tactical  information  from  the  multi-sensor  and  multiple  target 
environments. 

(2)  Positively  detect,  track,  and  identified  the  time  critical  targets  at  any  time  and  any  place 
under  all  weather  conditions. 

(3)  Accurately  deploy  weapon  system  for  surgical  strike  to  the  enemy  target. 

(4)  Preciously  discriminate  targets  from  friends  and  foes,  to  avoid  friendly  killing. 

Most  aircraft  development  involves  multiple  sensors.  Some  sensors  such  as  synthetic  aperture 
radar  can  provide  picture  like  images  at  very  long  range.  Sensors,  such  as  radar,  IFF,  electro¬ 
optic  cameras  and  infrared  images,  can  provide  tactical  information  to  detect,  track  and 
positively  identify  targets.  All  of  sensor  needed  to  integrated  in  order  to  generate  the  tactical 
information  for  the  smart  aircraft.  Integration  of  multiple  sensors  is  not  a  simple  task,  as  it 
requires  the  application  of  mathematical  concepts  and  computer  algorithms.  These  form  the 
bases  of  multi-sensor  information  fusion  technology,  which  directly  influences  the  development 
of  smart  aircraft. 

Multi-Sensor  Information  Fusion  (MSIF)  technology  includes  a  Multi-Sensor  Track  Fusion 
(MSTF)  model,  a  Multiple  Target  Identification  (MTIDF)  model  and  a  Multi-Sensor  Image 
Fusion  (MSIF)  model.  The  MSTF  model  creates  fused  tracks  from  the  multi-senor  trackers.  The 
MTIDF  model  creates  fused  target  identifications  from  the  multi- sensor  and  multiple  target 
identification  system.  The  MSIF  model  creates  the  fused  image  from  the  multi-sensor  image 
information  system. 

Multi-sensor  fusion  technology  is  the  tool  to  achieved  multi-sensor  integration.  Aircraft  without 
multi-sensor  fusion  information  technology  operates  less  effectively  on  the  battlefield.  Aircraft 
with  multi-sensor  information  fusion  technology  can  detect,  track  and  identified  the  time  critical 
targets  quickly  with  great  precision.  Smart  aircraft  with  multi-sensor  information  fusion 
technology  can  help  the  war  fighter  to  avoid  fratricide  and  help  the  aircrew  to  established 
superiority  in  the  air,  land  and  sea. 


2  MULTI-SENSOR  CORRELATION  (MSC)  PROCESSOR 


The  objective  of  the  Multi-Sensor  Correlation  (MSC)  processor  is  to  provide  the  logic  and 
algorithms  for  the  fusion  processor.  The  MSC  processor  estimates  the  linear  relationship 
between  two  given  feature  vectors  X  and  Y,  according  to  the  correlation  coefficient  between  X 
and  Y.  The  equation  to  calculate  the  correlation  coefficient  between  two  feature  vectors  X  and  Y, 
can  be  expressed  as  following: 


Let:  X  ={xi,  X2,  X3, . x,,}  be  the  feature  vector  of  an  object 

Y  ={yi,  y2,  ys, . v„ }  be  the  feature  vector  of  another  object 


Then:  the  correlation  coefficient  between  the  two  feature  vectors  X  and  Y  is: 

Cxy  =  X*Y  /  (X»X  -  X*Y  +  Y*Y)  (1)  [Jeun,  1997] 

Where:  Cxy  is  the  correlation  coefficient  between  feature  vectors  X  and  Y 
X*X  is  the  dot  product  of  X 
X* Y  is  the  dot  product  of  X  and  Y 
Y •  Y  is  the  dot  product  of  Y 

Decision  rules  concerning  the  correlation  coefficient  might  include  the  following: 

(1)  If  the  Coefficient  of  Correlation  (Cxy)  satisfies  the  following  condition:  0.95  <=  Cxy  <=1.0, 
then  X  and  Y  are  most  likely  correlated 

(2)  If  the  Coefficient  of  correlation  (Cxy)  satisfies  the  following  condition:  0.0  <=  Cxy<=0.95, 
then  X  and  Y  are  most  likely  uncorrelated. 

The  boundary  condition,  0.95,  is  selected  based  upon  the  feature  vector  elements’  precision. 

3  MULTI-SENSOR  TRACK  FUSION  (MSTF)  MODEL 


The  objective  of  the  Multi-Sensor  Track  Fusion  (MSTF)  model  is  to  create  the  fused  track  from 
the  multi-sensor  trackers.  Figure  1  depicts  a  Multi-Sensor  Track  Fusion  model  block  diagram. 


Figure  1:  The  Multiple-Sensor  Track  Fusion  Model  block  diagram  depicts  the  analysis  paths  that 
provide  the  Aircrew  with  meaningful  and  useful  knowledge. 


The  MSTF  model  automatically  generates  fused  tracks  in  the  mission  computers  for  the  aircrew 
to  make  quick  tactical  decisions.  The  fused  tracks  can  be  use  for  accurate  detection  of  objects  in 
the  battle  space. 

The  MSTF  model  consists  of  multiple  sensors,  trackers  and  multi-sensor  correlation  processor 
and  the  vehicle  interface  unit,  and  the  aircrew.  The  multiple  sensors  provide  sensor  reports  as  the 
input  vector  to  trackers,  and  the  trackers  generate  the  feature  vector  for  the  target.  The  feature 
vector  indicates  kinematics  parameters,  such  as  position  vector,  velocity  vector  and  acceleration 
vector  of  detected  targets.  The  multi-sensor  correlation  processor  is  use  to  compute  the 
correlation  coefficients  between  the  tracks.  The  multi-sensor  correlation  processor  also  used  to 
discriminate  tracks.  The  vehicle  interface  unit  (VIU)  serves  as  the  communication  between  the 
Multi-Sensor  Correlation  Processor  and  the  aircrew. 

The  output  of  the  trackers,  have  the  following  feature  elements  inside  each  feature  vector: 
{Latitude,  Longitude,  Bearing,  Target  Range,  Range  Rate,  Ground  Speed,  Course  or  Direction, 
Altitude}.  These  feature  elements  are  the  tactical  kinematics  parameters  that  can  characterize, 
detect  and  track  targets.  [Jeun,  Whittaker,  2002] 

4  EXAMPLE  OF  MSTF  MODEL 

Consider  that  the  multi-sensor  track  fusion  model  consists  of  only  two  sensors  and  one  is  a  radar 
sensor  (radar  tracker)  the  other  one  is  IFF  sensor  (IIF  tracker). 

Example  #1: 

Suppose  the  radar  tracker,  computes  a  feature  vector  for  track  #1,  denoted  as  Tl:  T1  =  {5.0, 
10.0,  75.0,  60.0,  2.0,  150.0,  75.0,  20.0}  and  suppose  the  IFF  tracker,  outputs  a  feature  vector  for 
the  track  #2,  denoted  as  T2:  T2  =  { 10.0,  40.0,  85.0,  65.0,  2.0,  140.0,  65.0,  85.0}.  The  correlation 
coefficient  between  the  two  feature  vectors  is  equal  to  0.87.  Therefore  the  result  of  the  fusion 
action  is  that  the  track  #1  and  track  #2  are  two  distinct  tracks. 

Example  #2: 

Suppose  the  radar  tracker,  produces  a  feature  vector  for  track  #1,  denoted  as  T3:  T3  =  {30.0, 
20.0,  60.0,  70.0,  2.0,100.0,  60.0,  30.0}  and  suppose  the  IFF  tracker,  defines  a  feature  vector  for 
the  track  #2,  denoted  as  T4:  T4  =  {30.0,  20.0,  60.0,  70.0,  2.0,  100.0,  60.0,  30.0}.  The  correlation 
coefficient  between  the  two  feature  vectors  is  equal  to  1.0.  Therefore  the  result  of  the  fusion 
action  is  that  track  #3  and  track  #4  most  likely  characterize  the  same  target 


5  MULTIPLE  TARGET  ID  LUSION  (MTIDE)  MODEL 


The  objective  of  the  Multiple  Target  ID  Fusion  (MTIDF)  model  is  to  provide  target  identification 
information  for  the  aircrew  to  make  accurate  decisions  concerning  target  selection  and  target 
prioritization. 


Figure  2:  Multiple  Target  Identification  Fusion  block  diagram  shows  the  analysis  paths  that 
provide  the  Aircrew  with  meaningful  and  useful  knowledge. 

The  aircrew  can  use  the  target  identification  information  from  the  MTIF  processor  to  accurately 
deploy  weapons  onto  enemy  targets.  Most  the  smart  aircraft  with  fusion  technology,  can  clearly 
determine  where  the  enemy  aircraft  are,  in  the  multi-sensor  and  multiple  target  environment. 
The  MTIDF  model  determines  the  fused  target  identification  from  the  multiple  targets. 

The  fused  target  probability  for  the  multi- sensor  and  multiple  targets  can  be 
expressed  as: 

P(Ai/Tk)/WTk) . P(SyTk) 

P(Tk/ S1.S2.S3 . Sn)  =  -  (2)  [Hall,  1992] 

xmyr,  )/w7j) . pcvt) 

Where  P(Tk/  S1.S2.S3 . Sn)  is  the  probability  of  target  Tk  given  in  multi-sensor 

environment. 

For  a  single  sensor  and  mulitple  targets,  the  fused  target  probability  reduces  to  the 
following: 


P(SfTk) 

P(Tk/S )  =  _  (3)  [Jeun,  1979] 

ms/Td 


This  expression  is  exactly  the  same  as  the  probability  of  association. 


6  EXAMPLE  OF  MTIDF  MODEL 


Figure  3  is  used  to  illustrate  the  different  views  of  two  targets,  T1  and  T2  as  seen  by  Sensor  #1, 
Sensor  #2  and  Sensor  #3.  The  basic  question  to  be  answered  can  be  formulated  as  the  following: 
“ Are  targets  #1  and  #2  observed  by  sensor  # 1 ,  sensor  #2  and  sensor  #. 3  the  same  target ”? 


Figure  3:  Potentially  two  targets  as  “seen”  by  three  different  sensors,  where  the  colored  regions 
indicate  the  fused  target  probability. 

Application  of  the  method  shown  in  [Jeun,  1995]  results  in  the  following  estimated  fused  target 
probabilities  for  the  target  set  presented  by  Figure  3.  The  estimated  fused  target  probability  of 
target#l  (Tl)  is  calculated  to  be  P(T1/S1,S2,  S3)  =  0.20  and  the  calculated  fused  probability  of 
target#2(T2)  is  estimated  to  be:  P(T2/S1,S2,S3)  =  0.80.  Because  the  fused  probability  of  T2  is 
greater  then  the  probability  of  Tl,  we  therefore  conclude  that  target  #1  and  target  #2  are  two 
distinct  targets. 


7  MULTI-SENSOR  IMAGE  FUSION  MODEL 


The  objective  of  the  Multi-Sensor  Image  Fusion  model  is  to  produce  consistent  images  from 
multiple  sensor  images.  The  Multi-Sensor  Image  Fusion  model  block  diagram  in  Figure  4  shows 


Figure  4:  Multi-Sensor  Image  Fusion  block  diagram  portrays  the  analysis  paths  that  provide  the 
Aircrew  with  meaningful  and  useful  knowledge. 

an  example  of  processes  that  can  be  employed.  The  Edge  Detector  method  and  Coherence 
Change  Detector  method,  and  Fourier  Descriptor  method  are  widely  used  image  analysis 
techniques.  [Delp,  Mitchell  and  Jain,  1986] 

Multiple  pictures,  like  images,  can  produce  information  overload,  making  it  impossible  to  be 
interpreted  by  human  mental  processing.  In  order  to  reduce  this  information  overload  requires 
some  automatic  integration,  or  better  yet,  application  of  the  multi- sensor  image  fusion  technique 
to  process  the  pictorial  images. 


8  EXAMPLE  OF  MSIF  MODEL 


To  help  understand  the  use  of  the  MSIF  model  consider  the  objects  pictured  in  Figures  5  and  6. 


Figure  5:  An  first  image  with  three  distinctly  shaped  and  colored  objects,  two  of  which  are 
different  from  Figure  6  and  one  of  which,  the  pale  blue  moon,  is  common  to  Figure  6. 


Figure  6:  A  second  image  with  three  distinctly  colored  and  shaped  objects,  two  of  which  are 
different  from  Figure  5  and  one  of  which,  the  pale  blue  moon,  is  common  to  Figure  5. 

Figure  5  contains  the  following  images:  (1)  A  red  star  with  feature  vector  Xi={0,  1,7,  0,  5,7, 
3,  5,  1,  3};  (2)  A  pale  blue  moon  with  feature  vector  X2  =  {1,  1,  1,  5,  5,  6,  7,  3,  3,  2};  and  (3)  A 
bright  yellow  sun  with  feature  vector  X3  =  {1,  1,  1,  7,  7,  5,  5,  4,  4,  3}.  Figure  6  portrays  the 
following  images:  (5)  A  dark  blue  triangle  with  feature  vector  Yi={  1,  1,  1,  1,  7,  7,  7,  7,  4,  4};  (6) 
A  pale  blue  moon  with  feature  vector  Y2  ={  1,  1,  1,  5,  5,  6,  7,  3,  3,  2};  and  (7)  A  bright  green 
cylinder  with  feature  vector  Y3  ={2,  2,  0,  0,  4,  4,  6,  6,  4,  4}.  X3,  X2,  X3,  Yi,  Y2  and  Y3  are 
obtained  applying  Friedman’s  code.  [Fu,  1982] 

The  question  that  needs  to  be  answered  is  the  following:  Can  an  Aircrew  flying  a  smart  aircraft, 
equipped  with  a  MSIF  model,  positively  identify  the  objects  contained  within  the  images 
presented  by  Figures  5  and  6? 


The  results  from  the  Multi-Sensor  Image  Fusion  model  are: 

(1)  The  fused  image  pale  blue  moon  appeared  in  both  Figure  5  and  Figure  6,  because  the 
correlation  coefficient  between  X2  and  Y2  is  1.0 

(2)  The  red  star  and  bright  yellow  sun  in  Figure  5  are  distinct  objects.  They  have  nothing  in 
common  with  other  images  in  Figure  5  and  Figure  6,  because: 

(a)  The  coefficient  of  correlation  between  Xi  and  X2  is  0.69 

(b)  The  coefficient  of  correlation  between  Xi  and  X3  is  0.54 

(c)  The  coefficient  of  correlation  between  X3  and  X2  is  0.91 

(d)  The  coefficient  of  correlation  between  Xi  and  Yi  is  0.69 

(e)  The  coefficient  of  correlation  between  Xi  and  Y2  is  0.58 

(f)  The  coefficient  of  correlation  between  Xi  and  Y3  is  0.58 

(g)  The  coefficient  of  correlation  between  X3  and  Y 1  is  0.77 

(h)  The  coefficient  of  correlation  between  X3  and  Y2  is  0.91 

(i)  The  coefficient  of  correlation  between  X3  and  Y3  is  0.66 

(3)  The  dark  blue  triangle  and  bright  green  cylinder  are  distinct  objects.  Again  they  have 
nothing  in  common  with  other  images  in  Figure  6  and  Figure  5,  because: 

(a)  The  coefficient  of  correlation  between  Y 1  and  Xi  is  0.69 

(b)  The  coefficient  of  correlation  between  Yi  and  X2  is  0.81 

(c)  The  coefficient  of  correlation  between  Y 1  and  X3  is  0.77 

(d)  The  coefficient  of  correlation  between  Y3  and  Xi  is  0.58 

(e)  The  coefficient  of  correlation  between  Y3  and  X2  is  0.73 

(f)  The  coefficient  of  correlation  between  Yi  and  Y2  is  0.81 

(g)  The  coefficient  of  correlation  between  Yi  and  Y3  is  0.88 

(h)  The  coefficient  of  correlation  between  Y3  and  X2  is  0.66 

9  CONCLUSIONS 

(1)  Multi-sensor  integration  is  widely  accepted  as  a  technology  that  is  ready  to  be  used 
in  product  development  and  production  environments. 

(2)  Personnel  need  to  become  knowledgeable  in  fusion  technology  so  that  it  too  can  be 
applied  in  these  environments. 

(3)  The  examples  of  the  Multi-Sensor  Track  Fusion  model,  the  Multiple  Target 
Identification  Fusion  model  and  the  Multi-Sensor  Image  Fusion  model  used 
simulated  information.  Nevertheless,  only  a  smart  aircraft  equipped  with  these 
models  can  quickly  and  positively  identify  the  objects  contained  with  the  images 
contained  in  Figure  5  and  Figure  6. 

(4)  The  fusion  techniques  outlined  in  this  paper  were  mathematically  proven.  However, 
further  testing  with  real  time  sensor  input  data  is  required  before  they  can  be  applied 
to  real  world  problems. 

(5)  The  fused  track,  fused  target  ID  and  fused  target  image  are  all  expressed  in  a  multi¬ 
dimensional  feature  vector.  The  algorithm  developer  and  the  software  engineer,  who 
work  with  fusion  technology,  need  to  be  knowledgeable  in  the  mathematical 
processing  techniques. 
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